Step-Ahead Error Feedback for Distributed Training with Compressed Gradient

Although the distributed machine learning methods can speed up training of large deep neural networks, communication cost has become non-negligible bottleneck to constrain performance. To address this challenge, gradient compression based communication-efficient were designed reduce cost, and more recently local error feedback was incorporated compensate for corresponding performance loss. However, in paper, we will show that a new "gradient mismatch" problem is raised by centralized lead degraded compared with full-precision training. solve critical problem, propose two novel techniques, 1) step ahead 2) averaging, rigorous theoretical analysis. Both our empirical results handle problem. The experimental even train faster common schemes than both regarding epochs without